Assessing Robustness of a Deep Learning Model for COVID-19 Classification on Chest Radiographs

The COVID-19 pandemic has made a dramatic impact on human life, medical systems, and financial resources. Due to the disease's pervasive nature, many different and interdisciplinary fields of research pivoted to study the disease. For example, deep learning (DL) techniques were employed early to assess patient diagnosis and prognosis from chest radiographs (CXRs) and computed tomography (CT) scans. While the use of artificial intelligence (AI) in the medical sector has displayed promising results, DL may suffer from lack of reproducibility and generalizability. In this study, the robustness of a pre-trained DL model utilizing the DenseNet-121 architecture was evaluated by using a larger collection of CXRs from the same institution that provided the original model with its test and training datasets. The current test set contained a larger span of dates, incorporated different strains of the virus, and included different immunization statuses. Considering differences in these factors, model performance between the original and current test sets was evaluated using area under the receiver operating characteristic curve (ROC AUC) [95% CI]. Statistical comparisons were performed using the Delong, Kolmogorov-Smirnov, and Wilcoxon rank-sum tests. Uniform manifold approximation and projection (UMAP) was used to help visualize whether underlying causes were responsible for differences in performance between test sets. In the task of classifying between COVID-positive and COVID-negative patients, the DL model achieved an AUC of 0.67 [0.65, 0.70], compared with the original performance of 0.76 [0.73, 0.79]. The results of this study suggest that underlying biases or overfitting may hinder performance when generalizing the model. © 2023 SPIE.

Deep learning to estimate covid-19 mortality risk from chest radiographs

Introduction: Management of coronavirus disease 2019 (COVID-19) requires accurate assessment of risk of future cardiopulmonary complications. Deep learning can extract complex relationships between medical imaging and clinical outcomes. Hypothesis: A deep learning model can predict 30-day mortality from COVID-19 based on a chest radiograph image. Methods: A deep learning model (CXR-CovRisk) was developed to estimate 30-day mortality risk using a single chest radiograph image (chest x-ray or CXR). The model was developed using 1,738 patients with PCR-confirmed coronavirus disease 2019 (COVID-19) from four Boston-area hospitals between March 1, 2020 and April 24, 2020. CXR-CovRisk was tested on 903 consecutive patients with confirmed COVID-19 between April 25, 2020 and June 15, 2020. CXR-CovRisk was compared to two published deep learning models (PXS and COVID-GMIC) and a clinical risk factor-based severity score for discrimination of 30-day mortality. The continuous risk score was converted to three risk groups: Low, Medium, and High based on development dataset probability quantiles. Results are provided for the independent testing set onlyResults: CXR-CovRisk had high discrimination for 30-day mortality (AUC = 0.839, 95% CI [0.79,0.89]), which was higher than when using a deep learning lung disease severity score (PXS AUC 0.750 [0.70,0.80], p < 0.001) or the output of a model trained for 96-hour mortality prediction (COVID-GMIC 0.755 [0.70,0.81], p = 0.003). CXR-CovRisk had added value to the clinical riskfactor based severity score (Clinical Severity Score AUC 0.799 [0.76,0.84] vs. Combined AUC 0.872 [0.84,0.90], p < 0.001). Among outpatients not admitted to the hospital, the CXR-CovRisk High-risk group had a high rate of subsequent hospital admission and 30-day mortality (composite event rate 11/26, 42.3%), higher than the medium-risk (30/179, 16.8%, p=0.005) and low-risk groups (17/172, 9.9%, p < 0.001). Conclusion: A deep learning model, CXR-CovRisk, can estimate 30-day mortality risk from a chest radiograph image.

Prolonged intubation in patients with prior cerebrovascular disease and COVID-19

Objective: To determine the odds of critical illness by day 28 and duration of mechanical ventilation (MV) over 45-day observation period in patients with history of cerebrovascular disease and COVID-19. Background: COVID-19-associated morbidity is correlated with multiple factors including age, comorbidities, and host response to the virus. Our understanding of the risk of critical illness due to prior neurological conditions remains limited. Here, we hypothesized that prior cerebrovascular disease is a risk factor for severe outcomes in COVID-19, including increased duration of MV. Design/Methods: A cross-sectional study of 1128 consecutive adult patients admitted to a tertiary care center in Boston, Massachusetts, and diagnosed with laboratory-confirmed COVID-19. The association between history of cerebrovascular disease and critical illness defined as MV or death was examined using logistic regression with inverse probability weighting of the propensity score. Cumulative incidence of successful extubation without death over 45 days was examined using competing risk analysis. Results: Of the 1128 adults admitted with COVID-19, 350 (36%) were critically ill by day 28. The median age was 59 years (standard deviation: 18 years), 640 (57%) were men, and 401 (36%) were Latinx ethnicity. As of June 2nd, 2020, 127 (11%) patients died. A total of 257 (23%) of patients had a prior neurological diagnosis;most common was cerebrovascular disease (16%). Prior cerebrovascular disease was significantly associated with critical illness (OR 1.54 [95% CI: 1.14 - 2.07]), lower rate of successful extubation (cause-specific HR 0.57 [95% CI: 0.33-0.98]), and increased duration of intubation (restricted mean time difference 4.02 days [95% CI: 0.34- 10.92]) compared to patients without cerebrovascular disease. Conclusions: History of cerebrovascular disease adversely affects COVID-19 outcomes including increased risk of critical illness and prolonged intubation. Further studies are needed to define measures that reduce risk of poor outcomes in this subpopulation.

Severity of Chest Imaging is Correlated with Risk of Acute Neuroimaging Findings among Patients with COVID-19.

BACKGROUND AND PURPOSE: Severe respiratory distress in patients with COVID-19 has been associated with higher rate of neurologic manifestations. Our aim was to investigate whether the severity of chest imaging findings among patients with coronavirus disease 2019 (COVID-19) correlates with the risk of acute neuroimaging findings. MATERIALS AND METHODS: This retrospective study included all patients with COVID-19 who received care at our hospital between March 3, 2020, and May 6, 2020, and underwent chest imaging within 10 days of neuroimaging. Chest radiographs were assessed using a previously validated automated neural network algorithm for COVID-19 (Pulmonary X-ray Severity score). Chest CTs were graded using a Chest CT Severity scoring system based on involvement of each lobe. Associations between chest imaging severity scores and acute neuroimaging findings were assessed using multivariable logistic regression. RESULTS: Twenty-four of 93 patients (26%) included in the study had positive acute neuroimaging findings, including intracranial hemorrhage (n = 7), infarction (n = 7), leukoencephalopathy (n = 6), or a combination of findings (n = 4). The average length of hospitalization, prevalence of intensive care unit admission, and proportion of patients requiring intubation were significantly greater in patients with acute neuroimaging findings than in patients without them (P < .05 for all). Compared with patients without acute neuroimaging findings, patients with acute neuroimaging findings had significantly higher mean Pulmonary X-ray Severity scores (5.0 [SD, 2.9] versus 9.2 [SD, 3.4], P < .001) and mean Chest CT Severity scores (9.0 [SD, 5.1] versus 12.1 [SD, 5.0], P = .041). The pulmonary x-ray severity score was a significant predictor of acute neuroimaging findings in patients with COVID-19. CONCLUSIONS: Patients with COVID-19 and acute neuroimaging findings had more severe findings on chest imaging on both radiographs and CT compared with patients with COVID-19 without acute neuroimaging findings. The severity of findings on chest radiography was a strong predictor of acute neuroimaging findings in patients with COVID-19.

Analysis of Stroke Detection during the COVID-19 Pandemic Using Natural Language Processing of Radiology Reports.

BACKGROUND AND PURPOSE: The coronavirus disease 2019 (COVID-19) pandemic has led to decreases in neuroimaging volume. Our aim was to quantify the change in acute or subacute ischemic strokes detected on CT or MR imaging during the pandemic using natural language processing of radiology reports. MATERIALS AND METHODS: We retrospectively analyzed 32,555 radiology reports from brain CTs and MRIs from a comprehensive stroke center, performed from March 1 to April 30 each year from 2017 to 2020, involving 20,414 unique patients. To detect acute or subacute ischemic stroke in free-text reports, we trained a random forest natural language processing classifier using 1987 randomly sampled radiology reports with manual annotation. Natural language processing classifier generalizability was evaluated using 1974 imaging reports from an external dataset. RESULTS: The natural language processing classifier achieved a 5-fold cross-validation classification accuracy of 0.97 and an F1 score of 0.74, with a slight underestimation (-5%) of actual numbers of acute or subacute ischemic strokes in cross-validation. Importantly, cross-validation performance stratified by year was similar. Applying the classifier to the complete study cohort, we found an estimated 24% decrease in patients with acute or subacute ischemic strokes reported on CT or MR imaging from March to April 2020 compared with the average from those months in 2017-2019. Among patients with stroke-related order indications, the estimated proportion who underwent neuroimaging with acute or subacute ischemic stroke detection significantly increased from 16% during 2017-2019 to 21% in 2020 (P = .01). The natural language processing classifier performed worse on external data. CONCLUSIONS: Acute or subacute ischemic stroke cases detected by neuroimaging decreased during the COVID-19 pandemic, though a higher proportion of studies ordered for stroke were positive for acute or subacute ischemic strokes. Natural language processing approaches can help automatically track acute or subacute ischemic stroke numbers for epidemiologic studies, though local classifier training is important due to radiologist reporting style differences.